{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c6a2f0f8-ce06-465b-aa02-496987f5a7c3",
   "metadata": {},
   "source": [
    "# Retrieval 模块代码实战"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d42c360-f39f-4ac5-81ad-160e5245a46e",
   "metadata": {},
   "source": [
    "文档加载器\n",
    "\n",
    "从许多不同的来源加载文档。LangChain提供了100多个不同的文档加载器，以及与其他主要提供商（如AirByte和Unstructured）的集成。我们提供了加载所有类型文档（HTML、PDF、代码）的集成，可以从各种位置（私有s3存储桶、公共网站）加载。\n",
    "\n",
    "文档转换器\n",
    "\n",
    "检索的一个关键部分是仅获取文档中相关的部分。这涉及到多个转换步骤，以最好地准备文档进行检索。其中一个主要步骤是将大型文档分割（或分块）为较小的块。LangChain提供了几种不同的算法来实现这一点，并且还针对特定文档类型（代码、Markdown等）优化了逻辑。\n",
    "\n",
    "文本嵌入模型\n",
    "\n",
    "检索的另一个关键部分是为文档创建嵌入。嵌入捕捉文本的语义含义，使您能够快速高效地找到其他相似的文本。LangChain与25多个不同的嵌入提供商和方法集成，从开源到专有的API，让您可以选择最适合您需求的嵌入方法。LangChain提供了一个标准接口，使您可以轻松切换模型。\n",
    "\n",
    "向量存储\n",
    "\n",
    "随着嵌入的兴起，出现了对数据库支持这些嵌入的高效存储和搜索的需求。LangChain与50多个不同的向量存储进行集成，从开源的本地存储到云托管的专有存储，让您可以选择最适合您需求的存储方式。LangChain提供了一个标准接口，使您可以轻松切换向量存储。\n",
    "\n",
    "检索器\n",
    "\n",
    "一旦数据存储在数据库中，您仍然需要检索它。LangChain支持许多不同的检索算法，这是我们增加最多价值的领域之一。我们支持一些简单易用的基本方法，例如简单的语义搜索。然而，我们还添加了一系列算法来提高性能。包括：\n",
    "\n",
    "父文档检索器：允许您为每个父文档创建多个嵌入，以便查找较小的块但返回较大的上下文。\n",
    "\n",
    "自查询检索器：用户的问题通常包含对不仅仅是语义的东西的引用，而是表达一些最好以元数据过滤器来表示的逻辑。自查询允许您从查询中解析出语义部分，并筛选出查询中存在的其他元数据过滤器。\n",
    "\n",
    "集合检索器：有时您可能希望从多个不同的来源或使用多个不同的算法检索文档。集合检索器使您能够轻松实现这一点。\n",
    "\n",
    "还有更多！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ee6f29f-8ba2-4bf7-ae80-bbab9987f2de",
   "metadata": {},
   "source": [
    "## 1. Document loaders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b94ea14-e803-4dbe-8c8e-c62b50a179b5",
   "metadata": {},
   "source": [
    "### 1.1 CSVLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "550561b5-b3e0-42e9-afc7-fbd7da548c3c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(page_content='Los Angeles: New York City\\n34°03′N: 40°42′46″N\\n118°15′W: 74°00′21″W', metadata={'source': 'data/data1.csv', 'row': 0}), Document(page_content='Los Angeles: Paris\\n34°03′N: 48°51′24″N\\n118°15′W: 2°21′03″E', metadata={'source': 'data/data1.csv', 'row': 1})]\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders.csv_loader import CSVLoader\n",
    "\n",
    "loader = CSVLoader(file_path=\"data/data1.csv\", encoding=\"utf-8\")\n",
    "\n",
    "data = loader.load()\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "567df017-44cf-4cf4-899e-f58e7c26acbf",
   "metadata": {},
   "source": [
    "### 1.2 TextLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "77350924-cf4c-436d-9405-5265cb9fa815",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='《消失的她》是一部充满悬疑和心理刺激的电影，讲述了丈夫何非的妻子李木子在结婚周年旅行中神秘失踪的故事。随后，出现了一个陌生女人冒充李木子，引发了一系列扑朔迷离的事件。以下是对该电影的详细解读：\\n\\n\\n剧情介绍与发展：\\n\\n电影一开始，丈夫何非报警称妻子李木子在国外旅行中突然失踪。然而，当警方介入调查后，情况变得异常复杂。一方面，陌生女人自称是李木子，并提供了身份证明和其他证据。另一方面，何非坚持认为这个女人并非自己的妻子。情节中呈现了众多的矛盾和谜团，让观众猜测真相。\\n\\n主要角色及其设定：\\n\\n何非（朱一龙饰）：失踪的妻子李木子的丈夫，他在失去妻子后竭尽全力寻找她。但他的神秘过去和心理问题让人产生怀疑，观众对他的真实性格和动机产生质疑。\\n\\n李木子（文咏珊饰）：消失的女主角，她在故事初期被认为是何非的妻子。然而，随着故事发展，她的真实身份和行为逐渐显露，让观众对她产生疑虑。\\n\\n陈麦（倪妮饰）：一名金牌律师，卷入了这起失踪案件中。她对案件产生浓厚兴趣，主动接手此案，并不断解密真相。她的出现为故事注入了更多的悬疑元素。\\n\\n林梅（黄子琪饰）：陈麦安排的替身女人，她是受害者之一。她与陈麦合作，试图揭示真相并向何非复仇。\\n\\n悬疑和心理刺激：\\n\\n电影的悬疑元素令人着迷。观众在故事中跟随何非、陈麦和李木子之间错综复杂的关系，推理和猜测真相。每个角色都有着不为人知的秘密和动机，令观众陷入扑朔迷离的氛围。随着剧情的发展，一层层真相被揭示，每一个细节都引发观众的疑虑和惊讶。\\n\\n同时，电影还探讨了人性、道德和正义等主题。何非的精神问题、李木子的复杂人格、陈麦的正义感，以及林梅等受害者的复仇心理，都使得故事更加丰富和引人深思。\\n\\n主题和意义：\\n\\n《消失的她》除了讲述一起失踪案件的悬疑故事，也强调了女性之间的团结与互助。陈麦和林梅是受害者的代表，她们联合其他受害者，通过巧妙的计划揭露何非的罪行，最终让他受到应有的惩罚。这展现了女性的力量和智慧，体现了女性在面对困境时的勇气和决心。\\n\\n同时，电影也警示人们珍惜生命中的幸福和美好时刻，对自己的行为负责，并追求自己的梦想。正义和尊严是电影所强调的核心价值观。\\n\\n综上所述，《消失的她》是一部充满悬疑和反转的电影，通过精心构建的剧情、角色设定和悬念设置，吸引观众的注意力并让他们对真相充满好奇。影片的主题也深刻探讨了女性的力量、正义和珍惜生命的价值观。这些元素共同构成了这部引人入胜的电影。', metadata={'source': '消失的她.txt'})]"
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.document_loaders import TextLoader\n",
    "\n",
    "docs = TextLoader('消失的她.txt', encoding=\"utf-8\").load()\n",
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d6222c80-afb6-426c-aaee-26ffb172bbd8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open('消失的她.txt', encoding=\"utf-8\") as f:\n",
    "    document = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fb291e54-7d03-4dcc-9732-0616c16c8b7b",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "《消失的她》是一部充满悬疑和心理刺激的电影，讲述了丈夫何非的妻子李木子在结婚周年旅行中神秘失踪的故事。随后，出现了一个陌生女人冒充李木子，引发了一系列扑朔迷离的事件。以下是对该电影的详细解读：\n",
      "\n",
      "\n",
      "剧情介绍与发展：\n",
      "\n",
      "电影一开始，丈夫何非报警称妻子李木子在国外旅行中突然失踪。然而，当警方介入调查后，情况变得异常复杂。一方面，陌生女人自称是李木子，并提供了身份证明和其他证据。另一方面，何非坚持认为这个女人并非自己的妻子。情节中呈现了众多的矛盾和谜团，让观众猜测真相。\n",
      "\n",
      "主要角色及其设定：\n",
      "\n",
      "何非（朱一龙饰）：失踪的妻子李木子的丈夫，他在失去妻子后竭尽全力寻找她。但他的神秘过去和心理问题让人产生怀疑，观众对他的真实性格和动机产生质疑。\n",
      "\n",
      "李木子（文咏珊饰）：消失的女主角，她在故事初期被认为是何非的妻子。然而，随着故事发展，她的真实身份和行为逐渐显露，让观众对她产生疑虑。\n",
      "\n",
      "陈麦（倪妮饰）：一名金牌律师，卷入了这起失踪案件中。她对案件产生浓厚兴趣，主动接手此案，并不断解密真相。她的出现为故事注入了更多的悬疑元素。\n",
      "\n",
      "林梅（黄子琪饰）：陈麦安排的替身女人，她是受害者之一。她与陈麦合作，试图揭示真相并向何非复仇。\n",
      "\n",
      "悬疑和心理刺激：\n",
      "\n",
      "电影的悬疑元素令人着迷。观众在故事中跟随何非、陈麦和李木子之间错综复杂的关系，推理和猜测真相。每个角色都有着不为人知的秘密和动机，令观众陷入扑朔迷离的氛围。随着剧情的发展，一层层真相被揭示，每一个细节都引发观众的疑虑和惊讶。\n",
      "\n",
      "同时，电影还探讨了人性、道德和正义等主题。何非的精神问题、李木子的复杂人格、陈麦的正义感，以及林梅等受害者的复仇心理，都使得故事更加丰富和引人深思。\n",
      "\n",
      "主题和意义：\n",
      "\n",
      "《消失的她》除了讲述一起失踪案件的悬疑故事，也强调了女性之间的团结与互助。陈麦和林梅是受害者的代表，她们联合其他受害者，通过巧妙的计划揭露何非的罪行，最终让他受到应有的惩罚。这展现了女性的力量和智慧，体现了女性在面对困境时的勇气和决心。\n",
      "\n",
      "同时，电影也警示人们珍惜生命中的幸福和美好时刻，对自己的行为负责，并追求自己的梦想。正义和尊严是电影所强调的核心价值观。\n",
      "\n",
      "综上所述，《消失的她》是一部充满悬疑和反转的电影，通过精心构建的剧情、角色设定和悬念设置，吸引观众的注意力并让他们对真相充满好奇。影片的主题也深刻探讨了女性的力量、正义和珍惜生命的价值观。这些元素共同构成了这部引人入胜的电影。\n"
     ]
    }
   ],
   "source": [
    "print(document)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 1.3 MarkdownLoader\n",
    "\n",
    "# !pip install markdown unstructured"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import UnstructuredMarkdownLoader\n",
    "markdown_path = \"data/README.md\"\n",
    "loader = UnstructuredMarkdownLoader(markdown_path)\n",
    "data = loader.load()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content=\"✨ Navigate at cookbook.openai.com\\n\\nExample code and guides for accomplishing common tasks with the OpenAI API. To run these examples, you'll need an OpenAI account and associated API key (create a free account here). Set an environment variable called OPENAI_API_KEY with your API key. Alternatively, in most IDEs such as Visual Studio Code, you can create an .env file at the root of your repo containing OPENAI_API_KEY=<your API key>, which will be picked up by the notebooks.\\n\\nMost code examples are written in Python, though the concepts can be applied in any language.\\n\\nFor other useful tools, guides and courses, check out these related resources from around the web.\\n\\nContributing\\n\\nThe OpenAI Cookbook is a community-driven resource. Whether you're submitting an idea, fixing a typo, adding a new guide, or improving an existing one, your contributions are greatly appreciated!\\n\\nBefore contributing, read through the existing issues and pull requests to see if someone else is already working on something similar. That way you can avoid duplicating efforts.\\n\\nIf there are examples or guides you'd like to see, feel free to suggest them on the issues page.\\n\\nIf you'd like to contribute new content, make sure to read through our contribution guidelines. We welcome high-quality submissions of new examples and guides, as long as they meet our criteria and fit within the scope of the cookbook.\\n\\nThe contents of this repo are automatically rendered into cookbook.openai.com based on registry.yaml.\", metadata={'source': 'data/README.md'})]"
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 1.4 PDFLoader"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import UnstructuredPDFLoader\n",
    "loader = UnstructuredPDFLoader(\"data/test.pdf\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "data = loader.load()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "data"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "id": "5641c6e2-0c8f-4963-8f60-eb3121d028e1",
   "metadata": {},
   "source": [
    "## 2. Document transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4d7e8f6-8f16-4cb3-ae47-6b61abe2c705",
   "metadata": {},
   "source": [
    "### 2.1 Recursively split by character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "46914ff2-6ab8-48e2-ab5a-83feb938920e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"data/test.txt\") as f:\n",
    "    testTXT = f.read()\n",
    "\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "chunk_size = 20,          # 块大小（每个分割文本的字符数量）\n",
    "chunk_overlap  = 10,       # 块重叠（两个相邻块之间重叠的字符数量）\n",
    "length_function = len,     # 长度函数（用于计算文本长度的函数）\n",
    "add_start_index = True,    # 添加起始索引（是否在结果中包含分割文本的起始索引）\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "1b675540-4456-4ba1-8fbe-ce5c02eb3368",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Madam Speaker, Madam' metadata={'start_index': 0}\n",
      "page_content='Madam Vice' metadata={'start_index': 15}\n"
     ]
    }
   ],
   "source": [
    "texts = text_splitter.create_documents([testTXT])\n",
    "print(texts[0])\n",
    "print(texts[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "64cc3508-de01-40a0-b9fb-87d27eb73e62",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(page_content='Madam Speaker, Madam', metadata={'start_index': 0}), Document(page_content='Madam Vice', metadata={'start_index': 15}), Document(page_content='Vice President,our', metadata={'start_index': 21}), Document(page_content='First Lady and', metadata={'start_index': 40}), Document(page_content='Lady and Second', metadata={'start_index': 46}), Document(page_content='Second Gentleman.', metadata={'start_index': 55}), Document(page_content='Members of Congress', metadata={'start_index': 74}), Document(page_content='Congress and the', metadata={'start_index': 85}), Document(page_content='and the Cabinet.', metadata={'start_index': 94}), Document(page_content='Justices of the', metadata={'start_index': 112}), Document(page_content='of the Supreme', metadata={'start_index': 121}), Document(page_content='Supreme Court. My', metadata={'start_index': 128}), Document(page_content='Court. My fellow', metadata={'start_index': 136}), Document(page_content='fellow Americans.', metadata={'start_index': 146}), Document(page_content='Last year COVID-19', metadata={'start_index': 165}), Document(page_content='COVID-19 kept us', metadata={'start_index': 175}), Document(page_content='kept us apart. This', metadata={'start_index': 184}), Document(page_content='This year we are', metadata={'start_index': 199}), Document(page_content='we are finally', metadata={'start_index': 209}), Document(page_content='finally together', metadata={'start_index': 216}), Document(page_content='together again.', metadata={'start_index': 224})]\n"
     ]
    }
   ],
   "source": [
    "print(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 2.2 Spilt Code"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "de18fe91-8234-4984-b5ca-8031b2afa984",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": "['cpp',\n 'go',\n 'java',\n 'kotlin',\n 'js',\n 'ts',\n 'php',\n 'proto',\n 'python',\n 'rst',\n 'ruby',\n 'rust',\n 'scala',\n 'swift',\n 'markdown',\n 'latex',\n 'html',\n 'sol',\n 'csharp',\n 'cobol',\n 'c',\n 'lua',\n 'perl']"
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.text_splitter import Language\n",
    "\n",
    "# 支持编程语言的完整列表\n",
    "[e.value for e in Language]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b0db0cdc-47a8-4d17-839c-6b20ae73db6e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "python_text = \"\"\"\n",
    "\n",
    " def print_multiplication_table():\n",
    "    for i in range(1, 10):\n",
    "        for j in range(1, i+1):\n",
    "            print(f'{j} * {i} = {i*j}\\t', end='')\n",
    "        print()\n",
    "\n",
    "print_multiplication_table()\n",
    "\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ecb3ddf7-5ff4-4763-aae6-1b926a75d08e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='def print_multiplication_table():'),\n Document(page_content='for i in range(1, 10):\\n        for j in range(1, i+1):'),\n Document(page_content=\"print(f'{j} * {i} = {i*j}\\t', end='')\"),\n Document(page_content='print()'),\n Document(page_content='print_multiplication_table()')]"
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "python_splitter = RecursiveCharacterTextSplitter.from_language(\n",
    "    language=Language.PYTHON, chunk_size=60, chunk_overlap=0\n",
    ")\n",
    "python_docs = python_splitter.create_documents([python_text])\n",
    "python_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "###2.3 Split by character"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "outputs": [],
   "source": [
    "with open(\"data/test.txt\") as f:\n",
    "    state_of_the_union = f.read()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "outputs": [],
   "source": [
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "\n",
    "text_splitter = CharacterTextSplitter(\n",
    "    separator=\"\\n\\n\",#指定了用于分割文本的分隔符为两个连续的换行符\n",
    "    chunk_size=20,#指定了每个分割后的文本块的大小为1000个字符\n",
    "    chunk_overlap=10,#指定了相邻文本块之间的重叠大小为200个字符\n",
    "    length_function=len,#指定了计算字符串长度的函数为内置的len函数\n",
    "    is_separator_regex=False,#指定了分隔符不是一个正则表达式\n",
    ")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 72, which is longer than the specified 20\n",
      "Created a chunk of size 36, which is longer than the specified 20\n",
      "Created a chunk of size 51, which is longer than the specified 20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Madam Speaker, Madam Vice President,our First Lady and Second Gentleman.'\n",
      "page_content='Members of Congress and the Cabinet.'\n"
     ]
    }
   ],
   "source": [
    "texts = text_splitter.create_documents([state_of_the_union])\n",
    "print(texts[0])\n",
    "print(texts[1])"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "id": "875f85ec-f8ac-4e0d-b3e8-ffa857155c81",
   "metadata": {},
   "source": [
    "## 3. Text embedding models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "771cbee1-0dd7-44b1-9621-64cd06e88dc4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_openai import OpenAIEmbeddings\n",
    "#from langchain.embeddings import OpenAIEmbeddings\n",
    "\n",
    "embeddings_model = OpenAIEmbeddings()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bd183ce6-482a-43a6-a24b-897515e26926",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": "(5, 1536)"
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 将一组文本转换为嵌入向量，并将结果存储在embeddings变量中\n",
    "embeddings = embeddings_model.embed_documents(\n",
    "    [\n",
    "        \"Hi there!\",\n",
    "        \"Oh, hello!\",\n",
    "        \"What's your name?\",\n",
    "        \"My friends call me World\",\n",
    "        \"Hello World!\"\n",
    "    ]\n",
    ")\n",
    "len(embeddings), len(embeddings[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8ab52c7f-2e32-4816-93a0-90b3f4882ca6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": "[0.005384807424727807,\n -0.0005522561790177147,\n 0.03896066510130955,\n -0.002939867294003909,\n -0.008987877434176603]"
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用embed_query方法将查询文本转换为嵌入向量，并将结果存储在embedded_query变量中\n",
    "embedded_query = embeddings_model.embed_query(\"What was the name mentioned in the conversation?\")\n",
    "embedded_query[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "192ba72b-566f-4feb-b2ff-37c9db17f707",
   "metadata": {},
   "source": [
    "## 4. vector store"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "# pip install chromadb"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "62ff545e-bda7-4887-9c31-82e403d8b4ea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders import TextLoader\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain.vectorstores import Chroma\n",
    "\n",
    "# 加载长文本\n",
    "raw_documents = TextLoader('消失的她.txt', encoding=\"utf-8\").load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e4b856b8-8c56-4220-b326-64290fdb10c5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 实例化文本分割器\n",
    "text_splitter = CharacterTextSplitter(chunk_size=100, chunk_overlap=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "361fefd2-cd44-4c7e-a6e2-c0895dbdb80d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 125, which is longer than the specified 100\n",
      "Created a chunk of size 112, which is longer than the specified 100\n",
      "Created a chunk of size 118, which is longer than the specified 100\n"
     ]
    }
   ],
   "source": [
    "# 分割文本\n",
    "documents = text_splitter.split_documents(raw_documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7e473f37-1fd7-415d-852f-4cfc099d5619",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 将分割后的文本，使用 OpenAI 嵌入模型获取嵌入向量，并存储在 Chroma 中\n",
    "db = Chroma.from_documents(documents, OpenAIEmbeddings())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "45f4c8a7-933d-42ac-8281-fd05d1b94726",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "《消失的她》是一部充满悬疑和心理刺激的电影，讲述了丈夫何非的妻子李木子在结婚周年旅行中神秘失踪的故事。随后，出现了一个陌生女人冒充李木子，引发了一系列扑朔迷离的事件。以下是对该电影的详细解读：\n"
     ]
    }
   ],
   "source": [
    "query = \"消失的她这部电影的角色有哪些？\"\n",
    "docs = db.similarity_search(query)\n",
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a05f5754-4b1d-4428-8936-4349f117d582",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "《消失的她》是一部充满悬疑和心理刺激的电影，讲述了丈夫何非的妻子李木子在结婚周年旅行中神秘失踪的故事。随后，出现了一个陌生女人冒充李木子，引发了一系列扑朔迷离的事件。以下是对该电影的详细解读：\n"
     ]
    }
   ],
   "source": [
    "\n",
    "embedding_vector = OpenAIEmbeddings().embed_query(query)\n",
    "docs = db.similarity_search_by_vector(embedding_vector)\n",
    "print(docs[0].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ce08af9-b0ed-48c1-8989-b8eb04022ee1",
   "metadata": {},
   "source": [
    "## 5. Retriever"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdd9c038-f99a-47f4-9500-2636c6a9e639",
   "metadata": {},
   "source": [
    "### 5.1 Ensemble Retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# 集合检索器（Ensemble Retriever）接受一组检索器作为输入，并将它们的get_relevant_documents()方法的结果进行集成，并根据互惠排序融合算法对结果进行重新排序。通过发挥不同算法的优势，集合检索器可以比单一算法获得更好的性能。\n",
    "\n",
    "# 最常见的模式是将稀疏检索器（如BM25）与密集检索器（如嵌入相似度）相结合，因为它们的优势互补。这也被称为“混合搜索”。稀疏检索器擅长基于关键词找到相关文档，而密集检索器擅长基于语义相似度找到相关文档。"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fd84169e-87f7-44a4-ad0c-191173acb770",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.retrievers import BM25Retriever, EnsembleRetriever\n",
    "from langchain.vectorstores import FAISS\n",
    "from langchain.embeddings import OpenAIEmbeddings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "6552a652-7f7b-45b9-9a88-cca54495a879",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "doc_list = [\n",
    "    \"I like apples\",\n",
    "    \"I like oranges\",\n",
    "    \"Apples and oranges are fruits\",\n",
    "]\n",
    "\n",
    "# initialize the bm25 retriever and faiss retriever\n",
    "## 稀松检索器\n",
    "bm25_retriever = BM25Retriever.from_texts(doc_list)\n",
    "bm25_retriever.k = 2\n",
    "\n",
    "## 密集检索器\n",
    "embedding = OpenAIEmbeddings()\n",
    "faiss_vectorstore = FAISS.from_texts(doc_list, embedding)\n",
    "faiss_retriever = faiss_vectorstore.as_retriever(search_kwargs={\"k\": 2})\n",
    "\n",
    "# initialize the ensemble retriever\n",
    "ensemble_retriever = EnsembleRetriever(\n",
    "    retrievers=[bm25_retriever, faiss_retriever], weights=[0.5, 0.5]\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2e5621e5-fd4b-4554-afcc-6f2ebddd452f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='I like apples'),\n Document(page_content='Apples and oranges are fruits')]"
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## 获取数据\n",
    "docs = ensemble_retriever.get_relevant_documents(\"apples\")\n",
    "docs"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 5.2 Self-querying"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "# 安装依赖 pip install --upgrade lark chromadb\n",
    "from langchain_community.vectorstores import Chroma\n",
    "from langchain_core.documents import Document\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "docs = [\n",
    "    Document(\n",
    "        page_content=\"A bunch of scientists bring back dinosaurs and mayhem breaks loose\",\n",
    "        metadata={\"year\": 1993, \"rating\": 7.7, \"genre\": \"science fiction\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Leo DiCaprio gets lost in a dream within a dream within a dream within a ...\",\n",
    "        metadata={\"year\": 2010, \"director\": \"Christopher Nolan\", \"rating\": 8.2},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea\",\n",
    "        metadata={\"year\": 2006, \"director\": \"Satoshi Kon\", \"rating\": 8.6},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"A bunch of normal-sized women are supremely wholesome and some men pine after them\",\n",
    "        metadata={\"year\": 2019, \"director\": \"Greta Gerwig\", \"rating\": 8.3},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Toys come alive and have a blast doing so\",\n",
    "        metadata={\"year\": 1995, \"genre\": \"animated\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"Three men walk into the Zone, three men walk out of the Zone\",\n",
    "        metadata={\n",
    "            \"year\": 1979,\"director\": \"Andrei Tarkovsky\",\"genre\": \"thriller\",\"rating\": 9.9},\n",
    "    ),\n",
    "]\n",
    "vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "from langchain.chains.query_constructor.base import AttributeInfo\n",
    "from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "metadata_field_info = [\n",
    "    AttributeInfo(\n",
    "        name=\"genre\",\n",
    "        description=\"The genre of the movie. One of ['science fiction', 'comedy', 'drama', 'thriller', 'romance', 'action', 'animated']\",\n",
    "        type=\"string\",\n",
    "    ),\n",
    "    AttributeInfo(\n",
    "        name=\"year\",\n",
    "        description=\"The year the movie was released\",\n",
    "        type=\"integer\",\n",
    "    ),\n",
    "    AttributeInfo(\n",
    "        name=\"director\",\n",
    "        description=\"The name of the movie director\",\n",
    "        type=\"string\",\n",
    "    ),\n",
    "    AttributeInfo(\n",
    "        name=\"rating\", description=\"A 1-10 rating for the movie\", type=\"float\"\n",
    "    ),\n",
    "]\n",
    "document_content_description = \"Brief summary of a movie\"\n",
    "llm = ChatOpenAI(temperature=0)\n",
    "retriever = SelfQueryRetriever.from_llm(\n",
    "    llm,\n",
    "    vectorstore,\n",
    "    document_content_description,\n",
    "    metadata_field_info,\n",
    ")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'director': 'Andrei Tarkovsky', 'genre': 'thriller', 'rating': 9.9, 'year': 1979}),\n Document(page_content='A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea', metadata={'director': 'Satoshi Kon', 'rating': 8.6, 'year': 2006})]"
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever.invoke(\"I want to watch a movie rated higher than 8.5\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='A bunch of normal-sized women are supremely wholesome and some men pine after them', metadata={'director': 'Greta Gerwig', 'rating': 8.3, 'year': 2019})]"
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever.invoke(\"Has Greta Gerwig directed any movies about women\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "retriever.invoke(\"What's a highly rated (above 8.5) science fiction film?\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "### 使用自查询检索器来指定 k：要获取的文档数"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "data": {
      "text/plain": "[Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'genre': 'science fiction', 'rating': 7.7, 'year': 1993}),\n Document(page_content='Toys come alive and have a blast doing so', metadata={'genre': 'animated', 'year': 1995})]"
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retriever = SelfQueryRetriever.from_llm(\n",
    "    llm,\n",
    "    vectorstore,\n",
    "    document_content_description,\n",
    "    metadata_field_info,\n",
    "    enable_limit=True,\n",
    ")\n",
    "\n",
    "# This example only specifies a relevant query\n",
    "retriever.invoke(\"What are two movies about dinosaurs\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### 5.3 MultiQueryRetriever"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "通过使用 LLM 从不同角度为给定的用户输入查询生成多个查询，可以自动执行提示调整过程。对于每个查询，它都会检索一组相关文档，并采用所有查询之间的唯一并集来获取更大的一组潜在相关文档"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:langchain.retrievers.multi_query:Generated queries: ['1. How can Task Decomposition be achieved through different methods?', '2. What strategies are commonly used for breaking down tasks into smaller components?', '3. What are the various techniques employed for Task Decomposition in practice?']\n"
     ]
    },
    {
     "data": {
      "text/plain": "[Document(page_content='Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n Document(page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\\nComponent One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\nTask Decomposition#', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n Document(page_content='Planning\\n\\nSubgoal and decomposition: The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks.\\nReflection and refinement: The agent can do self-criticism and self-reflection over past actions, learn from mistakes and refine them for future steps, thereby improving the quality of final results.\\n\\n\\nMemory', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"}),\n Document(page_content='(3) Task execution: Expert models execute on the specific tasks and log results.\\nInstruction:', metadata={'description': 'Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.\\nAgent System Overview In a LLM-powered autonomous agent system, LLM functions as the agent’s brain, complemented by several key components:', 'language': 'en', 'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'title': \"LLM Powered Autonomous Agents | Lil'Log\"})]"
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Build a sample vectorDB\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_community.vectorstores import Chroma\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "# Load blog post\n",
    "loader = WebBaseLoader(\"https://lilianweng.github.io/posts/2023-06-23-agent/\")\n",
    "data = loader.load()\n",
    "\n",
    "# Split\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
    "splits = text_splitter.split_documents(data)\n",
    "\n",
    "# VectorDB\n",
    "embedding = OpenAIEmbeddings()\n",
    "vectordb = Chroma.from_documents(documents=splits, embedding=embedding)\n",
    "\n",
    "\n",
    "from langchain.retrievers.multi_query import MultiQueryRetriever\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "\n",
    "llm = ChatOpenAI(temperature=0)\n",
    "retriever_from_llm = MultiQueryRetriever.from_llm(\n",
    "    retriever=vectordb.as_retriever(), llm=llm\n",
    ")\n",
    "\n",
    "# Set logging for the queries\n",
    "#import logging\n",
    "\n",
    "#logging.basicConfig()\n",
    "#logging.getLogger(\"langchain.retrievers.multi_query\").setLevel(logging.INFO)\n",
    "question = \"What are the approaches to Task Decomposition?\"\n",
    "unique_docs = retriever_from_llm.get_relevant_documents(query=question)\n",
    "#len(unique_docs)\n",
    "unique_docs"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "###自定义提示词和解析器"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.output_parsers import PydanticOutputParser\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "\n",
    "# Output parser will split the LLM result into a list of queries\n",
    "class LineList(BaseModel):\n",
    "    # \"lines\" is the key (attribute name) of the parsed output\n",
    "    lines: List[str] = Field(description=\"Lines of text\")\n",
    "\n",
    "\n",
    "class LineListOutputParser(PydanticOutputParser):\n",
    "    def __init__(self) -> None:\n",
    "        super().__init__(pydantic_object=LineList)\n",
    "\n",
    "    def parse(self, text: str) -> LineList:\n",
    "        lines = text.strip().split(\"\\n\")\n",
    "        return LineList(lines=lines)\n",
    "\n",
    "\n",
    "output_parser = LineListOutputParser()\n",
    "\n",
    "QUERY_PROMPT = PromptTemplate(\n",
    "    input_variables=[\"question\"],\n",
    "    template=\"\"\"You are an AI language model assistant. Your task is to generate five\n",
    "    different versions of the given user question to retrieve relevant documents from a vector\n",
    "    database. By generating multiple perspectives on the user question, your goal is to help\n",
    "    the user overcome some of the limitations of the distance-based similarity search.\n",
    "    Provide these alternative questions separated by newlines.\n",
    "    Original question: {question}\"\"\",\n",
    ")\n",
    "llm = ChatOpenAI(temperature=0)\n",
    "\n",
    "# Chain\n",
    "llm_chain = LLMChain(llm=llm, prompt=QUERY_PROMPT, output_parser=output_parser)\n",
    "\n",
    "# Other inputs\n",
    "question = \"What are the approaches to Task Decomposition?\""
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "outputs": [
    {
     "ename": "OutputParserException",
     "evalue": "Failed to parse LineList from completion 1. Got: 1 validation error for LineList\n  Input should be a valid dictionary or instance of LineList [type=model_type, input_value=1, input_type=int]\n    For further information visit https://errors.pydantic.dev/2.6/v/model_type",
     "output_type": "error",
     "traceback": [
      "\u001B[1;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[1;31mValidationError\u001B[0m                           Traceback (most recent call last)",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain_core\\output_parsers\\pydantic.py:33\u001B[0m, in \u001B[0;36mPydanticOutputParser._parse_obj\u001B[1;34m(self, obj)\u001B[0m\n\u001B[0;32m     32\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;28missubclass\u001B[39m(\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mpydantic_object, pydantic\u001B[38;5;241m.\u001B[39mBaseModel):\n\u001B[1;32m---> 33\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mpydantic_object\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mmodel_validate\u001B[49m\u001B[43m(\u001B[49m\u001B[43mobj\u001B[49m\u001B[43m)\u001B[49m\n\u001B[0;32m     34\u001B[0m \u001B[38;5;28;01melif\u001B[39;00m \u001B[38;5;28missubclass\u001B[39m(\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mpydantic_object, pydantic\u001B[38;5;241m.\u001B[39mv1\u001B[38;5;241m.\u001B[39mBaseModel):\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\pydantic\\main.py:509\u001B[0m, in \u001B[0;36mBaseModel.model_validate\u001B[1;34m(cls, obj, strict, from_attributes, context)\u001B[0m\n\u001B[0;32m    508\u001B[0m __tracebackhide__ \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mTrue\u001B[39;00m\n\u001B[1;32m--> 509\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mcls\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43m__pydantic_validator__\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mvalidate_python\u001B[49m\u001B[43m(\u001B[49m\n\u001B[0;32m    510\u001B[0m \u001B[43m    \u001B[49m\u001B[43mobj\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mstrict\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mstrict\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mfrom_attributes\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mfrom_attributes\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mcontext\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mcontext\u001B[49m\n\u001B[0;32m    511\u001B[0m \u001B[43m\u001B[49m\u001B[43m)\u001B[49m\n",
      "\u001B[1;31mValidationError\u001B[0m: 1 validation error for LineList\n  Input should be a valid dictionary or instance of LineList [type=model_type, input_value=1, input_type=int]\n    For further information visit https://errors.pydantic.dev/2.6/v/model_type",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001B[1;31mOutputParserException\u001B[0m                     Traceback (most recent call last)",
      "Cell \u001B[1;32mIn[57], line 7\u001B[0m\n\u001B[0;32m      2\u001B[0m retriever \u001B[38;5;241m=\u001B[39m MultiQueryRetriever(\n\u001B[0;32m      3\u001B[0m     retriever\u001B[38;5;241m=\u001B[39mvectordb\u001B[38;5;241m.\u001B[39mas_retriever(), llm_chain\u001B[38;5;241m=\u001B[39mllm_chain, parser_key\u001B[38;5;241m=\u001B[39m\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mlines\u001B[39m\u001B[38;5;124m\"\u001B[39m\n\u001B[0;32m      4\u001B[0m )  \u001B[38;5;66;03m# \"lines\" is the key (attribute name) of the parsed output\u001B[39;00m\n\u001B[0;32m      6\u001B[0m \u001B[38;5;66;03m# Results\u001B[39;00m\n\u001B[1;32m----> 7\u001B[0m unique_docs \u001B[38;5;241m=\u001B[39m \u001B[43mretriever\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mget_relevant_documents\u001B[49m\u001B[43m(\u001B[49m\n\u001B[0;32m      8\u001B[0m \u001B[43m    \u001B[49m\u001B[43mquery\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[38;5;124;43m\"\u001B[39;49m\u001B[38;5;124;43mWhat does the course say about regression?\u001B[39;49m\u001B[38;5;124;43m\"\u001B[39;49m\n\u001B[0;32m      9\u001B[0m \u001B[43m)\u001B[49m\n\u001B[0;32m     10\u001B[0m \u001B[38;5;28mlen\u001B[39m(unique_docs)\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain_core\\retrievers.py:245\u001B[0m, in \u001B[0;36mBaseRetriever.get_relevant_documents\u001B[1;34m(self, query, callbacks, tags, metadata, run_name, **kwargs)\u001B[0m\n\u001B[0;32m    243\u001B[0m \u001B[38;5;28;01mexcept\u001B[39;00m \u001B[38;5;167;01mException\u001B[39;00m \u001B[38;5;28;01mas\u001B[39;00m e:\n\u001B[0;32m    244\u001B[0m     run_manager\u001B[38;5;241m.\u001B[39mon_retriever_error(e)\n\u001B[1;32m--> 245\u001B[0m     \u001B[38;5;28;01mraise\u001B[39;00m e\n\u001B[0;32m    246\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[0;32m    247\u001B[0m     run_manager\u001B[38;5;241m.\u001B[39mon_retriever_end(\n\u001B[0;32m    248\u001B[0m         result,\n\u001B[0;32m    249\u001B[0m     )\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain_core\\retrievers.py:238\u001B[0m, in \u001B[0;36mBaseRetriever.get_relevant_documents\u001B[1;34m(self, query, callbacks, tags, metadata, run_name, **kwargs)\u001B[0m\n\u001B[0;32m    236\u001B[0m _kwargs \u001B[38;5;241m=\u001B[39m kwargs \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_expects_other_args \u001B[38;5;28;01melse\u001B[39;00m {}\n\u001B[0;32m    237\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_new_arg_supported:\n\u001B[1;32m--> 238\u001B[0m     result \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_get_relevant_documents(\n\u001B[0;32m    239\u001B[0m         query, run_manager\u001B[38;5;241m=\u001B[39mrun_manager, \u001B[38;5;241m*\u001B[39m\u001B[38;5;241m*\u001B[39m_kwargs\n\u001B[0;32m    240\u001B[0m     )\n\u001B[0;32m    241\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[0;32m    242\u001B[0m     result \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_get_relevant_documents(query, \u001B[38;5;241m*\u001B[39m\u001B[38;5;241m*\u001B[39m_kwargs)\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain\\retrievers\\multi_query.py:161\u001B[0m, in \u001B[0;36mMultiQueryRetriever._get_relevant_documents\u001B[1;34m(self, query, run_manager)\u001B[0m\n\u001B[0;32m    147\u001B[0m \u001B[38;5;28;01mdef\u001B[39;00m \u001B[38;5;21m_get_relevant_documents\u001B[39m(\n\u001B[0;32m    148\u001B[0m     \u001B[38;5;28mself\u001B[39m,\n\u001B[0;32m    149\u001B[0m     query: \u001B[38;5;28mstr\u001B[39m,\n\u001B[0;32m    150\u001B[0m     \u001B[38;5;241m*\u001B[39m,\n\u001B[0;32m    151\u001B[0m     run_manager: CallbackManagerForRetrieverRun,\n\u001B[0;32m    152\u001B[0m ) \u001B[38;5;241m-\u001B[39m\u001B[38;5;241m>\u001B[39m List[Document]:\n\u001B[0;32m    153\u001B[0m \u001B[38;5;250m    \u001B[39m\u001B[38;5;124;03m\"\"\"Get relevant documents given a user query.\u001B[39;00m\n\u001B[0;32m    154\u001B[0m \n\u001B[0;32m    155\u001B[0m \u001B[38;5;124;03m    Args:\u001B[39;00m\n\u001B[1;32m   (...)\u001B[0m\n\u001B[0;32m    159\u001B[0m \u001B[38;5;124;03m        Unique union of relevant documents from all generated queries\u001B[39;00m\n\u001B[0;32m    160\u001B[0m \u001B[38;5;124;03m    \"\"\"\u001B[39;00m\n\u001B[1;32m--> 161\u001B[0m     queries \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mgenerate_queries\u001B[49m\u001B[43m(\u001B[49m\u001B[43mquery\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mrun_manager\u001B[49m\u001B[43m)\u001B[49m\n\u001B[0;32m    162\u001B[0m     \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39minclude_original:\n\u001B[0;32m    163\u001B[0m         queries\u001B[38;5;241m.\u001B[39mappend(query)\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain\\retrievers\\multi_query.py:178\u001B[0m, in \u001B[0;36mMultiQueryRetriever.generate_queries\u001B[1;34m(self, question, run_manager)\u001B[0m\n\u001B[0;32m    167\u001B[0m \u001B[38;5;28;01mdef\u001B[39;00m \u001B[38;5;21mgenerate_queries\u001B[39m(\n\u001B[0;32m    168\u001B[0m     \u001B[38;5;28mself\u001B[39m, question: \u001B[38;5;28mstr\u001B[39m, run_manager: CallbackManagerForRetrieverRun\n\u001B[0;32m    169\u001B[0m ) \u001B[38;5;241m-\u001B[39m\u001B[38;5;241m>\u001B[39m List[\u001B[38;5;28mstr\u001B[39m]:\n\u001B[0;32m    170\u001B[0m \u001B[38;5;250m    \u001B[39m\u001B[38;5;124;03m\"\"\"Generate queries based upon user input.\u001B[39;00m\n\u001B[0;32m    171\u001B[0m \n\u001B[0;32m    172\u001B[0m \u001B[38;5;124;03m    Args:\u001B[39;00m\n\u001B[1;32m   (...)\u001B[0m\n\u001B[0;32m    176\u001B[0m \u001B[38;5;124;03m        List of LLM generated queries that are similar to the user input\u001B[39;00m\n\u001B[0;32m    177\u001B[0m \u001B[38;5;124;03m    \"\"\"\u001B[39;00m\n\u001B[1;32m--> 178\u001B[0m     response \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mllm_chain\u001B[49m\u001B[43m(\u001B[49m\n\u001B[0;32m    179\u001B[0m \u001B[43m        \u001B[49m\u001B[43m{\u001B[49m\u001B[38;5;124;43m\"\u001B[39;49m\u001B[38;5;124;43mquestion\u001B[39;49m\u001B[38;5;124;43m\"\u001B[39;49m\u001B[43m:\u001B[49m\u001B[43m \u001B[49m\u001B[43mquestion\u001B[49m\u001B[43m}\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mcallbacks\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mrun_manager\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mget_child\u001B[49m\u001B[43m(\u001B[49m\u001B[43m)\u001B[49m\n\u001B[0;32m    180\u001B[0m \u001B[43m    \u001B[49m\u001B[43m)\u001B[49m\n\u001B[0;32m    181\u001B[0m     lines \u001B[38;5;241m=\u001B[39m response[\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mtext\u001B[39m\u001B[38;5;124m\"\u001B[39m]\n\u001B[0;32m    182\u001B[0m     \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mverbose:\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain_core\\_api\\deprecation.py:145\u001B[0m, in \u001B[0;36mdeprecated.<locals>.deprecate.<locals>.warning_emitting_wrapper\u001B[1;34m(*args, **kwargs)\u001B[0m\n\u001B[0;32m    143\u001B[0m     warned \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mTrue\u001B[39;00m\n\u001B[0;32m    144\u001B[0m     emit_warning()\n\u001B[1;32m--> 145\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m wrapped(\u001B[38;5;241m*\u001B[39margs, \u001B[38;5;241m*\u001B[39m\u001B[38;5;241m*\u001B[39mkwargs)\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain\\chains\\base.py:378\u001B[0m, in \u001B[0;36mChain.__call__\u001B[1;34m(self, inputs, return_only_outputs, callbacks, tags, metadata, run_name, include_run_info)\u001B[0m\n\u001B[0;32m    346\u001B[0m \u001B[38;5;250m\u001B[39m\u001B[38;5;124;03m\"\"\"Execute the chain.\u001B[39;00m\n\u001B[0;32m    347\u001B[0m \n\u001B[0;32m    348\u001B[0m \u001B[38;5;124;03mArgs:\u001B[39;00m\n\u001B[1;32m   (...)\u001B[0m\n\u001B[0;32m    369\u001B[0m \u001B[38;5;124;03m        `Chain.output_keys`.\u001B[39;00m\n\u001B[0;32m    370\u001B[0m \u001B[38;5;124;03m\"\"\"\u001B[39;00m\n\u001B[0;32m    371\u001B[0m config \u001B[38;5;241m=\u001B[39m {\n\u001B[0;32m    372\u001B[0m     \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mcallbacks\u001B[39m\u001B[38;5;124m\"\u001B[39m: callbacks,\n\u001B[0;32m    373\u001B[0m     \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mtags\u001B[39m\u001B[38;5;124m\"\u001B[39m: tags,\n\u001B[0;32m    374\u001B[0m     \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mmetadata\u001B[39m\u001B[38;5;124m\"\u001B[39m: metadata,\n\u001B[0;32m    375\u001B[0m     \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mrun_name\u001B[39m\u001B[38;5;124m\"\u001B[39m: run_name,\n\u001B[0;32m    376\u001B[0m }\n\u001B[1;32m--> 378\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43minvoke\u001B[49m\u001B[43m(\u001B[49m\n\u001B[0;32m    379\u001B[0m \u001B[43m    \u001B[49m\u001B[43minputs\u001B[49m\u001B[43m,\u001B[49m\n\u001B[0;32m    380\u001B[0m \u001B[43m    \u001B[49m\u001B[43mcast\u001B[49m\u001B[43m(\u001B[49m\u001B[43mRunnableConfig\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43m{\u001B[49m\u001B[43mk\u001B[49m\u001B[43m:\u001B[49m\u001B[43m \u001B[49m\u001B[43mv\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;28;43;01mfor\u001B[39;49;00m\u001B[43m \u001B[49m\u001B[43mk\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mv\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;129;43;01min\u001B[39;49;00m\u001B[43m \u001B[49m\u001B[43mconfig\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mitems\u001B[49m\u001B[43m(\u001B[49m\u001B[43m)\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;28;43;01mif\u001B[39;49;00m\u001B[43m \u001B[49m\u001B[43mv\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;129;43;01mis\u001B[39;49;00m\u001B[43m \u001B[49m\u001B[38;5;129;43;01mnot\u001B[39;49;00m\u001B[43m \u001B[49m\u001B[38;5;28;43;01mNone\u001B[39;49;00m\u001B[43m}\u001B[49m\u001B[43m)\u001B[49m\u001B[43m,\u001B[49m\n\u001B[0;32m    381\u001B[0m \u001B[43m    \u001B[49m\u001B[43mreturn_only_outputs\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mreturn_only_outputs\u001B[49m\u001B[43m,\u001B[49m\n\u001B[0;32m    382\u001B[0m \u001B[43m    \u001B[49m\u001B[43minclude_run_info\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43minclude_run_info\u001B[49m\u001B[43m,\u001B[49m\n\u001B[0;32m    383\u001B[0m \u001B[43m\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain\\chains\\base.py:163\u001B[0m, in \u001B[0;36mChain.invoke\u001B[1;34m(self, input, config, **kwargs)\u001B[0m\n\u001B[0;32m    161\u001B[0m \u001B[38;5;28;01mexcept\u001B[39;00m \u001B[38;5;167;01mBaseException\u001B[39;00m \u001B[38;5;28;01mas\u001B[39;00m e:\n\u001B[0;32m    162\u001B[0m     run_manager\u001B[38;5;241m.\u001B[39mon_chain_error(e)\n\u001B[1;32m--> 163\u001B[0m     \u001B[38;5;28;01mraise\u001B[39;00m e\n\u001B[0;32m    164\u001B[0m run_manager\u001B[38;5;241m.\u001B[39mon_chain_end(outputs)\n\u001B[0;32m    166\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m include_run_info:\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain\\chains\\base.py:153\u001B[0m, in \u001B[0;36mChain.invoke\u001B[1;34m(self, input, config, **kwargs)\u001B[0m\n\u001B[0;32m    150\u001B[0m \u001B[38;5;28;01mtry\u001B[39;00m:\n\u001B[0;32m    151\u001B[0m     \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_validate_inputs(inputs)\n\u001B[0;32m    152\u001B[0m     outputs \u001B[38;5;241m=\u001B[39m (\n\u001B[1;32m--> 153\u001B[0m         \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43m_call\u001B[49m\u001B[43m(\u001B[49m\u001B[43minputs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mrun_manager\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mrun_manager\u001B[49m\u001B[43m)\u001B[49m\n\u001B[0;32m    154\u001B[0m         \u001B[38;5;28;01mif\u001B[39;00m new_arg_supported\n\u001B[0;32m    155\u001B[0m         \u001B[38;5;28;01melse\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_call(inputs)\n\u001B[0;32m    156\u001B[0m     )\n\u001B[0;32m    158\u001B[0m     final_outputs: Dict[\u001B[38;5;28mstr\u001B[39m, Any] \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mprep_outputs(\n\u001B[0;32m    159\u001B[0m         inputs, outputs, return_only_outputs\n\u001B[0;32m    160\u001B[0m     )\n\u001B[0;32m    161\u001B[0m \u001B[38;5;28;01mexcept\u001B[39;00m \u001B[38;5;167;01mBaseException\u001B[39;00m \u001B[38;5;28;01mas\u001B[39;00m e:\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain\\chains\\llm.py:104\u001B[0m, in \u001B[0;36mLLMChain._call\u001B[1;34m(self, inputs, run_manager)\u001B[0m\n\u001B[0;32m     98\u001B[0m \u001B[38;5;28;01mdef\u001B[39;00m \u001B[38;5;21m_call\u001B[39m(\n\u001B[0;32m     99\u001B[0m     \u001B[38;5;28mself\u001B[39m,\n\u001B[0;32m    100\u001B[0m     inputs: Dict[\u001B[38;5;28mstr\u001B[39m, Any],\n\u001B[0;32m    101\u001B[0m     run_manager: Optional[CallbackManagerForChainRun] \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m,\n\u001B[0;32m    102\u001B[0m ) \u001B[38;5;241m-\u001B[39m\u001B[38;5;241m>\u001B[39m Dict[\u001B[38;5;28mstr\u001B[39m, \u001B[38;5;28mstr\u001B[39m]:\n\u001B[0;32m    103\u001B[0m     response \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mgenerate([inputs], run_manager\u001B[38;5;241m=\u001B[39mrun_manager)\n\u001B[1;32m--> 104\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mcreate_outputs\u001B[49m\u001B[43m(\u001B[49m\u001B[43mresponse\u001B[49m\u001B[43m)\u001B[49m[\u001B[38;5;241m0\u001B[39m]\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain\\chains\\llm.py:258\u001B[0m, in \u001B[0;36mLLMChain.create_outputs\u001B[1;34m(self, llm_result)\u001B[0m\n\u001B[0;32m    256\u001B[0m \u001B[38;5;28;01mdef\u001B[39;00m \u001B[38;5;21mcreate_outputs\u001B[39m(\u001B[38;5;28mself\u001B[39m, llm_result: LLMResult) \u001B[38;5;241m-\u001B[39m\u001B[38;5;241m>\u001B[39m List[Dict[\u001B[38;5;28mstr\u001B[39m, Any]]:\n\u001B[0;32m    257\u001B[0m \u001B[38;5;250m    \u001B[39m\u001B[38;5;124;03m\"\"\"Create outputs from response.\"\"\"\u001B[39;00m\n\u001B[1;32m--> 258\u001B[0m     result \u001B[38;5;241m=\u001B[39m [\n\u001B[0;32m    259\u001B[0m         \u001B[38;5;66;03m# Get the text of the top generated string.\u001B[39;00m\n\u001B[0;32m    260\u001B[0m         {\n\u001B[0;32m    261\u001B[0m             \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39moutput_key: \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39moutput_parser\u001B[38;5;241m.\u001B[39mparse_result(generation),\n\u001B[0;32m    262\u001B[0m             \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfull_generation\u001B[39m\u001B[38;5;124m\"\u001B[39m: generation,\n\u001B[0;32m    263\u001B[0m         }\n\u001B[0;32m    264\u001B[0m         \u001B[38;5;28;01mfor\u001B[39;00m generation \u001B[38;5;129;01min\u001B[39;00m llm_result\u001B[38;5;241m.\u001B[39mgenerations\n\u001B[0;32m    265\u001B[0m     ]\n\u001B[0;32m    266\u001B[0m     \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mreturn_final_only:\n\u001B[0;32m    267\u001B[0m         result \u001B[38;5;241m=\u001B[39m [{\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39moutput_key: r[\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39moutput_key]} \u001B[38;5;28;01mfor\u001B[39;00m r \u001B[38;5;129;01min\u001B[39;00m result]\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain\\chains\\llm.py:261\u001B[0m, in \u001B[0;36m<listcomp>\u001B[1;34m(.0)\u001B[0m\n\u001B[0;32m    256\u001B[0m \u001B[38;5;28;01mdef\u001B[39;00m \u001B[38;5;21mcreate_outputs\u001B[39m(\u001B[38;5;28mself\u001B[39m, llm_result: LLMResult) \u001B[38;5;241m-\u001B[39m\u001B[38;5;241m>\u001B[39m List[Dict[\u001B[38;5;28mstr\u001B[39m, Any]]:\n\u001B[0;32m    257\u001B[0m \u001B[38;5;250m    \u001B[39m\u001B[38;5;124;03m\"\"\"Create outputs from response.\"\"\"\u001B[39;00m\n\u001B[0;32m    258\u001B[0m     result \u001B[38;5;241m=\u001B[39m [\n\u001B[0;32m    259\u001B[0m         \u001B[38;5;66;03m# Get the text of the top generated string.\u001B[39;00m\n\u001B[0;32m    260\u001B[0m         {\n\u001B[1;32m--> 261\u001B[0m             \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39moutput_key: \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43moutput_parser\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mparse_result\u001B[49m\u001B[43m(\u001B[49m\u001B[43mgeneration\u001B[49m\u001B[43m)\u001B[49m,\n\u001B[0;32m    262\u001B[0m             \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mfull_generation\u001B[39m\u001B[38;5;124m\"\u001B[39m: generation,\n\u001B[0;32m    263\u001B[0m         }\n\u001B[0;32m    264\u001B[0m         \u001B[38;5;28;01mfor\u001B[39;00m generation \u001B[38;5;129;01min\u001B[39;00m llm_result\u001B[38;5;241m.\u001B[39mgenerations\n\u001B[0;32m    265\u001B[0m     ]\n\u001B[0;32m    266\u001B[0m     \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mreturn_final_only:\n\u001B[0;32m    267\u001B[0m         result \u001B[38;5;241m=\u001B[39m [{\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39moutput_key: r[\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39moutput_key]} \u001B[38;5;28;01mfor\u001B[39;00m r \u001B[38;5;129;01min\u001B[39;00m result]\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain_core\\output_parsers\\pydantic.py:61\u001B[0m, in \u001B[0;36mPydanticOutputParser.parse_result\u001B[1;34m(self, result, partial)\u001B[0m\n\u001B[0;32m     57\u001B[0m \u001B[38;5;28;01mdef\u001B[39;00m \u001B[38;5;21mparse_result\u001B[39m(\n\u001B[0;32m     58\u001B[0m     \u001B[38;5;28mself\u001B[39m, result: List[Generation], \u001B[38;5;241m*\u001B[39m, partial: \u001B[38;5;28mbool\u001B[39m \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mFalse\u001B[39;00m\n\u001B[0;32m     59\u001B[0m ) \u001B[38;5;241m-\u001B[39m\u001B[38;5;241m>\u001B[39m TBaseModel:\n\u001B[0;32m     60\u001B[0m     json_object \u001B[38;5;241m=\u001B[39m \u001B[38;5;28msuper\u001B[39m()\u001B[38;5;241m.\u001B[39mparse_result(result)\n\u001B[1;32m---> 61\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43m_parse_obj\u001B[49m\u001B[43m(\u001B[49m\u001B[43mjson_object\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[1;32mD:\\Program Files (x86)\\python3\\lib\\site-packages\\langchain_core\\output_parsers\\pydantic.py:42\u001B[0m, in \u001B[0;36mPydanticOutputParser._parse_obj\u001B[1;34m(self, obj)\u001B[0m\n\u001B[0;32m     37\u001B[0m             \u001B[38;5;28;01mraise\u001B[39;00m OutputParserException(\n\u001B[0;32m     38\u001B[0m                 \u001B[38;5;124mf\u001B[39m\u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mUnsupported model version for PydanticOutputParser: \u001B[39m\u001B[38;5;130;01m\\\u001B[39;00m\n\u001B[0;32m     39\u001B[0m \u001B[38;5;124m                    \u001B[39m\u001B[38;5;132;01m{\u001B[39;00m\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mpydantic_object\u001B[38;5;241m.\u001B[39m\u001B[38;5;18m__class__\u001B[39m\u001B[38;5;132;01m}\u001B[39;00m\u001B[38;5;124m\"\u001B[39m\n\u001B[0;32m     40\u001B[0m             )\n\u001B[0;32m     41\u001B[0m     \u001B[38;5;28;01mexcept\u001B[39;00m (pydantic\u001B[38;5;241m.\u001B[39mValidationError, pydantic\u001B[38;5;241m.\u001B[39mv1\u001B[38;5;241m.\u001B[39mValidationError) \u001B[38;5;28;01mas\u001B[39;00m e:\n\u001B[1;32m---> 42\u001B[0m         \u001B[38;5;28;01mraise\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_parser_exception(e, obj)\n\u001B[0;32m     43\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:  \u001B[38;5;66;03m# pydantic v1\u001B[39;00m\n\u001B[0;32m     44\u001B[0m     \u001B[38;5;28;01mtry\u001B[39;00m:\n",
      "\u001B[1;31mOutputParserException\u001B[0m: Failed to parse LineList from completion 1. Got: 1 validation error for LineList\n  Input should be a valid dictionary or instance of LineList [type=model_type, input_value=1, input_type=int]\n    For further information visit https://errors.pydantic.dev/2.6/v/model_type"
     ]
    }
   ],
   "source": [
    "# Run\n",
    "retriever = MultiQueryRetriever(\n",
    "    retriever=vectordb.as_retriever(), llm_chain=llm_chain, parser_key=\"lines\"\n",
    ")  # \"lines\" is the key (attribute name) of the parsed output\n",
    "\n",
    "# Results\n",
    "unique_docs = retriever.get_relevant_documents(\n",
    "    query=\"What does the course say about regression?\"\n",
    ")\n",
    "len(unique_docs)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
